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Summative evaluation of Cycles R-40 of the Listen Look Learn (LLL) Multi-Media 
Communication Skills System was conducted in 1967-68. using 28 experimental (LLL 
system) and 28 control (variety of basal reading programs) classes of first-grade 
children. Testing included the use of the Metropolitan Readiness Tests; the 
Pintner-Cunningham Primary Test; the Cooperative Primary Tests. Listening; and the 
Stanford Achievement Test. Additional evaluation, provided through teacher 
questionnaires, indicated enthusiasm for the LLL system. Analysis of variance and 
analysis of covariance, using the BMD05V Computer Program, indicated significantly 
higher reading achievement test scores for LLL children, similar scores for 
average-ability LLL children and high-ability control children, and higher scores for 
suburban children than for urban children. It was concluded that the LLL system is 
preferred by teachers and is extremely effective with children at all ability levels, 
from all community sizes, and from every socioeconomic status. Tables of results are 
included. (M0) 



' 



/ / 



7 



if 



LT\ 



rvJ 

eg 

K\ 



IU 



ERLC 

mMBBmu 

4U- 



“■'“■nsrs.(S'"‘ . 

THIS DOCUMENT Sfcfcti OPINIONS 

"“*£ nTKsTniiy Ks%t -official office of education-. 

POSITION OR POLICY. 



An Experimental Study Designed to Test the Relative Effectiveness of 

a Multi-Media Instructional System 



Presented By 
Donald R. Senter, Ed.D 
Educational Developmental Laboratories 



At The 

1969 AERA Conference 



Prepared By 
Research Department 

Educational Developmental Laboratories 



*0 

is. 

00 



u 



1 






4 







! 



! 






\ 



i 




Introduction i 

During the period 1965-67, extensive formative research was done with the j 

Listen took Learn (LLL) Multi-Media Communication Skills System. Continuous 
revision and improvement was conducted as a result of the feedback obtained. 

Summarive evaluation of Cycles R-40 in first-grade classrooms was conducted 
during the 1967-68- school year. Two major questions were asked in this evaluation. J 
The question of the effectiveness of the LLL system when compared with traditional J 
basal reading programs was primary. Effectiveness of the program was defined, for 
the purpose of this study, to be the achievement levels attained by students in 
LLL and control classes. Secondly, it is recognized that the success of any i 

instructional program is largely dependent upon the opinion of teachers regarding jj 
its effectiveness and the confidence the teachers have in the ability of the 
• program to satisfactorily aid them in achieving their objectives. 

Twenty-eight experimental (LLL system) and twenty-eight control (other basal j 
reading programs) classes of first-grade children constituted the sample. 

! 

The experimental classes in all cases used the Listen Look Learn system, - j 

Cycles R-40, produced by Educational Developmental Laboratories, Inc. and the • '< 

L. W. Singer Company. The Ginn Basic Reading Program was used by twelve of the j 

twenty-eight control groups. Five control groups used Scott Foresman; two groups . 
used Houghton Mifflin; two groups used Harper S- Row; two groups used S.R.A.; and 
Webster Programmed Reading, Open Court Reading Program, Bill Martin Sounds of l 

Language, Banks Street Readers, and Economy Series were used by one control group j 

each as the basis of the language arts program. For the purpose of this study, I 

all basal reading programs were classified as the control groups. 1 

The tests used for the objective evaluation of the 1967-68 study were: 

1. Metropolitan Readiness Tests - administered in October 1967 ‘{ 

2. Pintner-Cunningham Primary Test - administered in October 1967 

3. Cooperative Primary Tests, Listening - administered in October 1967. ! 

and May 1968 j 

4. Stanford Achievement Test - administered in May 1968 

Subjective Evaluation 1 

Examination of subjective responses to questionnaires submitted by the twenty- 
eight teachers who used the LLL system was extremely favorable. Twenty-two teachers j 
responded that they would like to use the LLL system the following year, and . 
twenty-one teachers judged the total system as superior to any basal programs with | 

which they had had experience. In the opinion of the majority of the teachers the \ 

growth patterns of the children in their classes was Improved in such areas as j 

listening accuracy, independent activity in small groups, and interest in reading 
and learning. Most cooperating teachers were able to handle the system successfully ] 
with minimal assistance or no assistance. The strength of the program most frequently 
mentioned by teachers related to the high interest level achieved and maintained 
throughout the year by the students. Seven of the twenty-eight responding teachers 
were particularly pleased with the interest exhibited by boys in the system. j 
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Analysis of Variance and Analysis of Covariance was done utilizing the BMDQ5V 
Computer Program. The primary analysis involved a 2 x 3 x 2 (treatment group x 
ability level x sex) factorial design. Two auxiliary Analyses of Covariance were 
performed. The first was a 2 x 4 x 2 (treatment group x size of community x sex) 
factorial design and the second was a 2 x 3 x 2 (treatment group x socioeconomic 
status x sex) factorial design. IQ was used as a covarlate for each of these 
designs. Main effects and interactions of main effects were considered. 



Because of the nature of the experiment, intact classroom units had to be used 
Analysis of Covariance was used to remove initial differences between groups. 



Table I gives the results of the analysis of the data using the Stanford 
Achievement Test subtests and the Cooperative Primary Test . Listening as the 
dependent variables. Children who had participated in the LLL system scored 
significantly higher on these tests than did children from the control group 
who used other basal reading programs. Ability level was a significant factor 
in achievement as would be expected. Sex differences were significant and 
favored girls. The only significant interactions found were treatment by 
ability level . 



TABLE I 
about here 



Two specific points are of interest in Figure 1 which shows the adjusted 
mean values for the Word Meaning subtest. First, average ability children who 
were in classes using the LLL system scored at almost the_same level ( x a 3*2l.56) 
as children of high ability who were in control classes 21 *64) . Secondly 

the slope of the line that shows the difference between LLL ind control groups* 
is much more pronounced for the average group. 



* 



Figure 1 
about here 



Figure 2 is a graphical representation of the results obtained on the 
Vocabulary subtest. • 



Figure 2 
about here 



A significant difference was found to exist at the .05 level that favored 
boys and girls from the LLL group over boys and girls from the control group for 
each part score from the Stanford Achievement Test and for the Cooperative Primary 
Test, Listening. Boys categorized as average (IQ range of 88 to 112) for the L 
purpose of this study, scored significantly higher on each of the five posttests 

and this difference was at a highly significant (p < . 01 ) level for three of the 
five tests. 



In summary, results favoring the students from LLL groups are consistent. 

No comparison of adjusted cell means showed results in favor of the control group 
The results for boys were of particular interest since boys often present greater 
problems with respect to achievement of reading skills in the early grades 
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Auxi liary Analyses 

\n order to evaluate the effectiveness of the ILL program in communities of 
varying sizes, an auxiliary analysis of the data was performed following non- 
stratified blocking on size of community from which the student came. Sizes of 
communities were grouped into four categories. 

* * 

Table || gives a summary of the F-ratios obtained from computation of the 
Analysis of Covariance where IQ scores were used as the covariate. For all 
subtests and the listening test, differences favoring the LLL group were highly 
significant. Significant differences were also found on scores according to 
size of communi ty. 

The two categories of size which were favored were the communities of less 
than 10,000. Communities of this size range were rural in a few cases but 
generally they would be classified as suburban. 

No differences due to sex were found to exist from this analysis. 

The only significant interaction that resulted from the analysis was treatment 
group (LLL or control) by size of community for the Paragraph Meaning subtest. It 
can be assumed that this significant interaction could have occurred by chance. 

With a total of thirty-five comparisons it would be expected that more than one 
interaction might occur by chance at the .05 level. 

TABLE II 
about here 

The second auxiliary analysis was performed fol lowing non-strati fied blocking 
on socioeconomic level. It was considered to be of major concern to determine 
what effect this classification might have on student achievement. Table III gives 
a summary of the F-ratios obtained from computation of the Analysis of Covariance 
where IQ scores were used as the covariate. Treatment group (LLL or control) was 
a significant main effect for all subtests and the listening test favoring the 
LLL students. Socioeconomic status of the student was a significant main effect 
for all subtests of the Stanford Achievement Test but not for the listening test. 

s. 

TABLE III 
about here 

Conclusions 

Subjective evaluation, collected from cooperating teachers and summarized by 
the EDL Research Department, indicated that the majority of the teachers who used 
the LLL system were extremely pleased with the effectiveness and operation of the 
instruments, interest level and greater vocabulary of the LLL materials, and the 
overall results achieved by their students. 
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The objective evaluation was consistent and results strongly favored the LLL 
system. The primary analysis which was done, following blocking on ability level, 
indicated that children of each ability level using the LLL system were able to * J 
achieve at a higher level than the control children using a basal program. In j 

thirteen of fifteen achievement test comparisons, this difference was a significant 1 
one in favor of the LLL group; two comparisons were non-significant. The inter- 
action of ti eatment group (LLL or control) and ability level with achievement was 
significant for this analysis. In general, this significant interaction was ! 

attributable to the excellent results of the average ability children who had been 
in LLL groups. ' 

if 

Auxiliary analyses were performed to test the effectiveness of the LLL system I 
in varying sizes of communities and with the three general classifications of j 

socioeconomic status. In both of the auxiliary analyses achievement exhibited by \ 
students from LLL groups was consistently superior to that exhibited by control 
students. Children from small communities, essentially suburban areas, were j 

favored as might be suggested a priori. A second finding, also presupposed by i 

experience, indicated that the high and average socioeconomic groups were able 
to achieve at a higher level. The interesting, and not predetermined, results 
of the two auxiliary analyses were that children from any of the size categories j 

involved and children from every socioeconomic classification performed consistently i 
better following use of the LLL system. 

The results of the year-long evaluative study indicate that the LLL. system j 

is a system that can be managed- by teachers, that teachers enjoy using and want 
to use again, and that it is an extremely effective teaching system for children 
of all ability levels, particularly the average ability range, from all sizes of j 

communities, and from every socioeconomic status. • | 

i 

To the extent to which teachers and children in this investigation are -j 

representative of first-grade teachers and pupils, and to the extent to which the 
evaluation instruments are valid and reliable, the following major conclusions are 
drawn from the data in this study. j 
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TABLE I 



Summary of F-Ratios and Error Mean Squares for the Analysis of Covariance 3 * for 
the Primary. Des i gn Prepared to Test Differences Resulting from the Ability Level 
of Students for Total Sample. 



Stanford Achievement Test 



Cooperative 



Source of Variation 


df 


Word 

Meaning 


Paragraph 

Meaning 


Vocab- 
u 1 a ry 


Word Study 
Skills 


Li stening 
Test 


T re a t men t 


1 


24.33** 


24.43** 


7 . 28 **. 


5.51** 


21.41** 


Ability Level * 


2 


21.1 6** 


20.63** 


21 .07** 


12.08** 


27 . 00 ** 


Sex 


1 


13.18** 


28 . 32** 


.37 


14.04** 


6:41* 


Treatment by Ability 


2 


9.72** 


8.08** 


3.38 


4.59** 


1.56 


Treatment by Sex 


1 


2.88 


.19 


.10 


1.49 


.90 


Ability by Sex 


2 


1.66 


1.13 


1.51 


1.05 


.71 


Treatment, by Ability by Sex 


2 


.33 


1.20 


.46 


1.57 


.10 


Error Mean Square 


933 


38.96 












929 




63.14 










880 






30.68 




r 




882 








67.53 






909 










24.35 



a. 

b. 



Metropolitan Reading Readiness Test was used as the covariate for this analysis 
Pi ntner- C unn ingham Primary Test scores were used to determine the ability level ^ 



4 % 



p < .05 (probability of this result occurring by chance is less than five times 
over one hundred replications) 

p < .01 (probability of this result occurring by chance is less than one time 
over one hundred replications) 



-5- 



4 



4 









) * 



! 

i 

i 



ip 

1 



j ^ 



i'fo 

It 

it • 



■ *■' ' • 4T’ V, 



ADJUSTED 

MEAN 

VALUES 




Graphical Representation of Two-Way Interaction 
of Ability Level by Treatment Group for Word Meaning Sobtest 

Figure 1 
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Graphical Representation of Two-Way Interaction 
of Ability Level by Treatment Group for Vocabulary Subtest 



Figure 2 
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TABLE II 



Summary of F-Ratios and Error Mean Squares for the Analysis of Covariance 3 * for 
the Auxiliary Design Prepared to Test Differences Resulting from Community Size 



Stanford Achievement Test 



Cooperative 



* 

.. / 

Source of Variation ' 


df 


Word 

Meaning 


Paragraph 

Meaninq 


Vocab- 
ul a ry 


Word Study 
Skills 


Listening 

Test 


Treatment 


l 


25.35** 


20.69** 


37.29** 


12.10** 


14.32** 


Size of. Community 


3 


11.66** 


14.02** 


21.03** 


11.00** 


9.63** 


Sex 


l 


.03 ' 


1.09 


1.82 


1.01 


1.17 


Treatment by Size 


3 


2.32 


5.17* 


2.79 


1.00 


1.93 


T reatment by Sex 


1 


1.07 


1.13 


3.46 


3.43 


2.68 


Size by Sex 


3 


3.26 


2.85 


2.32 


3.33 


2.06 


Treatment by Size by Sex 


3 


1.49 


1.89 


1.02 


1.08 


.97 


L 

Error Mean Square • 


984 


40.38 


42.74 


36*94 


49.17 


43.91 



a. 

b. 

* 

** 



Pintner-Cunnlngham IQ, score was used as the covariate for this analysis. 
Degrees of Freedom for Word Meaning Subtest. 

p<.05 (probability of this result occurring by chance is less than five times 
over one hundred replications) 

p < . 01 (probability of this result occurring by chance Is less than one time 
. over one hundred replications) 




TABLE III 



Summary of F-Rotios and Error Mean Squares for the Analysis of Covariance 5 ** for the 
Auxiliary Design Prepared to Test Differences Resulting from Socioeconomic Background 
of Community 1 . 
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Stanford Achievement Test 


Cooperative 


• 

• 

Source of Variation 


df 


Word 

Meaning 


Paragraph 

Meaninq 


Vocab- 

ulary 


Word Study 
Skills 


Li stening 
Test . 


Treatment 


1 


19.37** 


12.46** 


16.31** 


18.19** 


6.3.1* 


Socioeconomic Status 


2 


11.01** 


9.27** 


12.69* 

* 


14.92** ' 


2.83 . 


Sex 


1 


2.91 


1.83 


2.13 


2.70 


1.06 


Treatment by Socioeconomic 
Status 


2 


1.83 


• .43 


1.47 


1.61 


2.32 


Treatment by Sex 


i 


2.1$ 


1.17 


.86 


1.50 


1.27 


Socioeconomic Status by Sex 


2 


.96 


0 

CO 

• 


1.33 


2.1! 


1.64 


Treatment by Socioeconomic 
Status by Sex 


2 


1.34 


2.06 


1.84 


.99 


2.06 


Error Mean Square 


916 


26.87 


rs. 

(n 

• 

CO 

c*\ 


35.41 


29.87 


43.28 



a. Pintner-Cunningham IQ, score was used as the covariate for this analysis. 

* p < .05 (probability of this result occurring by chance is less than five times 

over one hundred replications). - 

** P<* 01 (probability of this result occurring by chance is less than one time 

over one hundred repl I cat ions) 
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